When an observer measures the angle to a star (including the sun), the sextant is held at some height above sea level. Since the Earth curves away at the sea's horizon, the sea horizon is necessarily down from the observers eye. Figure 1 is an exagerated view of this concept. When the DIP is obtained from a Nautical almanac it includes a correction for refraction, which is not shown here. That correction is usually very small as long as the celestial object is more than about $35^\circ$ high - on the order of 0.3 minutes.
This article will explain how to compute the DIP angle.
Given Earth, with radius R=6371 km center, C at (0,0). For now, we can work
in two dimensions. Suppose that an observer is on top of a mountain that is 2 km high.
At point P, which marks the horizon for our observer, there is a right triangle. We know Earths radius,
$R=6371,$ and we know that the observer is 2 km further up, so $h=6373$ km. By Pathagoras, we can calculate "d".
$$d=\sqrt{h^2-R^2}.$$
Having found "d", we can use trigonometry to say that $\alpha = tan^{-1}\frac{R}{d}$
There are many ways to establish coordinates for point P, but one of the easiest is to establish new basis vectors.
$$\mathbf{u}=\frac{A-C}{|A-C|}$$
$$\mathbf{e_1 = u} \qquad \mathbf{e_2=\left(y(u), -x(u)\right)}$$
Build the vector with angle $\alpha$.
$$\mathbf{v} = cos(\alpha)\cdot\mathbf{e_1}+sin(\alpha)\cdot\mathbf{e_2}$$
Then, $P=A-d\cdot\mathbf{v}.$
Now suppose we use a sextant to measure the angle to star B. Using our horizon, not the "Level Horizon Line", but rather the observer horizon we measure an angle of $58^\circ 29.16'$
We want to calculate the DIP as a function of the observers height above sea level. In the figure, sea level would be the "Level Horizon Line". Since we have already found 'P', which is the Earth horizon edge that can be seen from our elevation, and since we know the observer's height, we can imagine a Level line at the observer's height and use that to compute the DIP angle.
Let vector $\ell_1 = \binom{-1}{0},$ which is the same as
$$\ell_1 = \frac{\binom{P_x-A_x}{A_y-A_y}}{\left|\binom{P_x}{A_y}-\binom{A_x}{A_y} \right|}$$
Let vector $\ell_2 = (P-A) = \frac{\binom{P_x}{P_y}-\binom{A_x}{A_y}}{\left| \binom{P_x}{P_y}-\binom{A_x}{A_y}\right|}$
Now we can use the law of cosines for vectors.
$$\ell_1\cdot\ell_2 = |\ell_1|\cdot|\ell_2|\cdot\cos\theta$$
to get the cosine of $\theta$ which here is the DIP angle. So,
$$\angle DIP = \cos^{-1}\frac{\ell_1\cdot\ell_2}{|\ell_1|\cdot|\ell_2|}$$
Throughout all of this, it is good to remember that our height above sea level is
$$A_y - \text{ Earth's Radius}.$$